Display control device, display control method, and program

ABSTRACT

An apparatus comprising a memory storing instructions and a control unit is disclosed. The control unit may be configured to execute the instructions to: acquire an image of real space, detect an object of interest within the real space image, obtain a still image of the object. and display a virtual image relating to the object superimposed on the still image.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a national phase entry under 35 U.S.C. §371of International Application No. PCT/JP2012/001963 filed Mar. 22, 2012,published on Oct. 4, 2012, as WO 2012/132334 A1, which claims priorityfrom Japanese Patent Application No. JP 2011-078072 filed in theJapanese Patent Office on Mar. 31, 2011.

TECHNICAL FIELD

The present disclosure relates to a display control device, a displaycontrol method, and a program.

BACKGROUND ART

Image recognition technology has become more advanced recently, enablingrecognition of the position or posture of a real object (e.g. an objectsuch as a sign board or building) contained in an input image from animaging device. As one application example of such object recognition,AR (Augmented Reality) application is known. According to the ARapplication, it is possible to superimpose a virtual object (e.g.advertising information, navigation information, or information for agame) associated with a real object onto the real object contained in areal space image. Note that the AR application is disclosed in PatentLiterature 1, for example.

CITATION LIST Patent Literature

PTL 1

JP 2010-238098A

SUMMARY Technical Problem

When a user uses the AR application with a mobile terminal having animaging function, the user can maintain the state where the virtualobject is superimposed on the real space image by pointing the imagingdirection of the mobile terminal toward the real object. However, it isburdensome for the user to maintain pointing the imaging directiontoward the real object.

In light of the foregoing, the present disclosure proposes a novel andimproved display control device, display control method and program thatenable improvement of usability for a user.

Solution to Problem

According to an embodiment of the present disclosure, there is providedAn apparatus including, a memory storing instructions; and a controlunit configured to execute the instructions to: acquire an image of realspace; detect an object of interest within the real space image; obtaina still image of the object; and display a virtual image relating to theobject superimposed on the still image.

Further, according to another embodiment of the present disclosure,there is provided A method including: acquiring a plurality of images ofreal space; detecting an object of interest within the plurality ofimages; obtaining a still image of the object; and displaying a virtualimage relating to the object superimposed on the still image.

Further, according to another embodiment of the present disclosure,there is provided A tangibly embodied non-transitory computer-readablemedium storing instructions which, when executed by a processor, performa method including acquiring a plurality of images of real space;detecting an object of interest within the plurality of images;obtaining a still image of the object; and displaying a virtual imagerelating to the object superimposed on the still image.

Advantageous Effects of Invention

As described above, the display control device, the display controlmethod and the program according to embodiments of the presentdisclosure enable improvement of usability for a user.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view showing a configuration of an Augmented Reality (AR)system according to the present disclosure;

FIG. 2 is a view showing a example of a real-time recognition screen;

FIG. 3 is a view showing a hardware configuration of a mobile terminal;

FIG. 4 is a block diagram showing a configuration of a mobile terminalaccording to a first embodiment;

FIG. 5 is a view showing layouts of a real-time recognition screen and astill image operation screen;

FIG. 6 is a view showing an alternative example of a still imageoperation screen;

FIG. 7 is a flowchart showing an operation of a mobile terminal inreal-time recognition mode;

FIG. 8 is a flowchart showing an operation of a mobile terminal in stillimage operation mode;

FIG. 9 is a block diagram showing a configuration of a mobile terminalaccording to a second embodiment;

FIG. 10 is a block diagram showing a configuration of a mobile terminalaccording to a third embodiment;

FIG. 11 is a view showing an AR system according to a fourth embodiment;

FIG. 12 is a functional block diagram showing configurations of a mobileterminal and an AR server according to the fourth embodiment;

FIG. 13 is a sequence chart showing a first operation example of thefourth embodiment;

FIG. 14 is a sequence chart showing a second operation example of thefourth embodiment; and

FIG. 15 is a sequence chart showing a third operation example of thefourth embodiment;

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

In this specification and the drawings, structural elements havingsubstantially the same function are distinguished by a differentalphabetical letter affixed to the same reference numeral in some cases.However, when there is no particular need to distinguish between aplurality of structural elements having substantially the same function,they are denoted only by the same reference numeral.

Further, “Description of Embodiments” will be provided in the followingorder.

1. Overview of AR System

2. Description of Embodiments

2-1. First Embodiment

2-2. Second Embodiment

2-3. Third Embodiment

2-4. Fourth Embodiment

3. Summary

Aspects of the present invention include an apparatus comprising amemory storing instructions and a control unit. The control unit may beconfigured to execute the instructions to: acquire an image of realspace, detect an object of interest within the real space image, obtaina still image of the object. and display a virtual image relating to theobject superimposed on the still image.

Aspects of the present invention include a method comprising acquiring aplurality of images of real space, detecting an object of interestwithin the plurality of images, and obtaining a still image of theobject. The method may further comprise displaying a virtual imagerelating to the object superimposed on the still image.

Aspects of the present invention include a tangibly embodiednon-transitory computer-readable medium storing instructions which, whenexecuted by a processor, perform a method, The method may compriseacquiring a plurality of images of real space, detecting an object ofinterest within the plurality of images, obtaining a still image of theobject, and displaying a virtual image relating to the objectsuperimposed on the still image.

1. Overview of AR System

The AR system according to the present disclosure may be implemented invarious embodiments as described in detail in “2-1. First Embodiment” to“2-4. Fourth Embodiment” by way of illustration. Further, the AR systemaccording to each embodiment may include a display control device(mobile terminal 20, AR server 30) including:

A. a control unit (241-244) that may cause display of a display deviceto transition between a real space moving image containing a movingimage of a real space and a, for example, composite image containing astill image of the real space and a virtual object corresponding to areal object contained in the still image.

Hereinafter, a basic configuration of the AR system common to eachembodiment is described firstly with reference to FIGS. 1 and 2.

FIG. 1 is a view showing a configuration or an AR system according tothe present disclosure. As shown in FIG. 1, the AR system according tothe disclosure includes a mobile terminal 20 having an imaging function.The mobile terminal 20 can, for example, capture a real space image anddisplay a virtual object corresponding to a real object contained in thereal space image superimposed onto the real space image on a display 26.The real space image may include, for example, a plurality of images ofreal space. When the real space image includes a plurality of images ofreal space obtained periodically, the real space image is said to be a“live image.” It is to be understood that the term “real space image,”as used herein, encompasses a single still image of real space, aplurality of still images of real space, and a live image of real space.

For example, when the real object is a station board 40 as shown in FIG.1, the mobile terminal 20 can capture a real space image containing thestation board 40 and display a real-time recognition screen in which avirtual object corresponding to the station board 40 is superimposed onthe display 26. The real-time recognition screen is specificallydescribed hereinafter with reference to FIG. 2.

FIG. 2 is a view showing a example of the real-time recognition screen.As shown in FIG. 2, a real-time recognition screen P contains display ofthe station board 40, which is a real object, and a time table 52, whichis a virtual object superimposed onto the station board 40. By lookingat the real-time recognition screen in which the virtual object issuperimposed, a user can obtain information which is not obtainable fromthe real space (in the case of FIG. 2, information about train scheduletime, which is not written on the station board 40).

Further, a user can change display of the virtual object contained inthe real-time recognition screen by conducting a predetermined operationon the display screen. For example, when an upward or downward slideoperation is conducted by a user on the real-time recognition screen Pshown in FIG. 2, the mobile terminal 20 may slide the subject of displayin the time table 52 upward or downward.

Furthermore, when a plurality of virtual objects is associated with onereal object, the mobile terminal 20 may switch the virtual object to bedisplayed among the plurality of virtual objects according to a useroperation. For example, when an leftward or rightward slide operation isconducted by a user on the real-time recognition screen P shown in FIG.2, the mobile terminal 20 may switch the virtual object to besuperimposed onto the station board 40 to an area map or a route map.

Note that, although a smart phone is shown as an example of the mobileterminal 20 in FIG. 1, the mobile terminal 20 is not limited to thesmart phone. For example, the mobile terminal 20 may be a PDA (PersonalDigital Assistant), a mobile telephone, a portable music playbackdevice, a portable video processing device, or a portable game machine.Further, the mobile terminal 20 is just an example of a display controldevice, and the display control device may be a server installed on thenetwork side or in any other suitable configuration.

Further, although the station board 40 is shown as an example of thereal object in FIGS. 1 and 2, the real object is not limited to thestation board 40. For example, the real object may be the object ofinterest or, for example, the object may be a sign board, a building, avehicle, a food product or a toy, or an identification pattern such as aQR code or an AR marker.

Incidentally, the above-described AR application that superimposes avirtual object onto a real space image is often, but not exclusively,executed by recognizing the captured real space image in real time.Therefore, a user can maintain the real-time recognition screen in whichthe virtual object is superimposed onto the real space image by pointingthe imaging direction of the mobile terminal 20 toward the real objectand maintaining the orientation for some period of time. However, it canbe burdensome for the user to maintain pointing the imaging direction ofthe mobile terminal 20 toward the real object. Further, in the statewhere the imaging direction of the mobile terminal 20 is pointed towardthe real object, it is difficult to conduct an operation for changingdisplay of the virtual object.

Given such and other circumstances, embodiments of the presentdisclosure have been invented. According to the embodiments of thedisclosure, it is possible to improve the usability and operability ofthe mobile terminal 20. Hereinafter, after the hardware configuration ofthe mobile terminal 20 is described, the embodiments of the presentdisclosure are sequentially described in detail.

Hardware Configuration of Mobile Terminal

FIG. 3 is an explanatory view showing the hardware configuration of themobile terminal 20. As shown in FIG. 3, the mobile terminal 20 includesa CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM(Random Access Memory) 203, an input device 208, an output device 210, astorage device 211, a drive 212, an imaging device 213, and acommunication device 215.

The CPU 201 may serve both as a processing device and a control device,and it controls the overall operations in the mobile terminal 20according to various kinds of programs. The CPU 201 may be amicroprocessor. The ROM 202 stores a program to be used by the CPU 201,processing parameters and the like. The RAM 203 may temporarily stores aprogram to be used in the execution on the CPU 201, parameters that varyin the execution and the like. The CPU 201, the ROM 202 and the RAM 203are connected through the host bus, which may be, for example, a CPU busand the like.

The input device 208 may be composed of an input means for a user toinput information, such as a mouse, a keyboard, a touch panel, a button,a microphone, a switch or a lever, and an input control circuit thatgenerates an input signal based on a user input and outputs it to theCPU 201, for example. A user of the mobile terminal 20 manipulates theinput device 208 to thereby input various kinds of data or direct aprocessing operation to the mobile terminal 20.

The output device 210 includes a display device such as a LCD (LiquidCrystal Display) device, an OLED (Organic Light Emitting Diode) deviceor a lamp. Further, the output device 210 includes a sound output devicesuch as a speaker or a headphone. For example, the display devicedisplays captured images, generated images and the like. On the otherhand, the sound output device may convert sound data and the like intosound and outputs the sound.

The storage device 211 is a device whose functions may include datastorage that may be configured as an example of a storage unit of themobile terminal 20 according to the embodiment. The storage device 211may include a storage medium, a recording device that records data intothe storage medium, a reading device that reads data from the storagemedium, a deleting device that deletes data recorded in the storagemedium and the like. The storage device 211 may store a program to beexecuted by the CPU 201 and various data.

The drive 212 is a reader/writer for a storage medium, and it may bebuilt in the mobile terminal 20 or attached externally. The drive 212may read information that is recorded on a removable storage medium 24such as a magnetic disk, an optical disk, a magneto-optical disk or asemiconductor memory which is attached thereto, and outputs theinformation to the RAM 203. Further, the drive 212 may write informationinto the removable storage medium 24.

The imaging device 213 may include an imaging optical system such as ataking lens or a zoom lens that condenses light, and a signal convertersuch as a CCD (Charge Coupled Device) or a CMOS (Complementary MetalOxide Semiconductor). The imaging optical system condenses light emittedfrom a subject and forms a subject image on the signal converter, andthe signal converter converts the formed subject image into anelectrical image signal.

The communication device 215 is a communication interface composed of acommunication device or the like to make a connection with the network12, for example. The communication device 215 may be a communicationdevice compatible with a wireless LAN (Local Area Network), acommunication device compatible with LTE (Long Term Evolution), or awired communication device that makes wired communication.

Note that the network 12 may be a wired or wireless transmission channelof information transmitted from a device connected with the network 12.For example, the network 12 may include a public network such as theInternet, a telephone line network or a satellite communicationsnetwork, various kinds of LAN (Local Area Network) including Ethernet(registered trademark), WAN (Wide Area Network) and the like. Further,the network 12 may include a leased line network such as IP-VPN(Internet Protocol-Virtual Private Network).

2. Description of Embodiments

An exemplary configuration of the AR system according to the presentdisclosure is described above with reference to FIGS. 1 to 3.Hereinafter, a first embodiment to a fourth embodiment according to thepresent disclosure are described in detail with reference to FIGS. 4 to15.

2-1. First Embodiment

(Configuration of Mobile Terminal)

FIG. 4 is a functional block diagram showing a configuration of a mobileterminal 20-1 according to the first embodiment. As shown in FIG. 4, themobile terminal 20-1 according to the first embodiment includes adisplay 26, a touch panel 27, an imaging device 213, a feature value DB220, an image recognition unit 224, a virtual object DB 228, a virtualobject acquisition unit 232, and a control unit 241.

The display 26 may include a display module composed of LCD, OLED or thelike. The display 26 displays various screens according to control bythe control unit 241. For example, the display 26 displays a real-timerecognition screen that contains a moving image of a real space beingcaptured by the imaging device 213, a still image operation screen,which is a composite image of a still image of the real space and avirtual object corresponding to a real object contained in the stillimage and the like. Note that, however, the real-time recognition screenis just an example of a real space moving image containing a movingimage of a real space, and the real space moving image may contain amoving image of a real space captured prior to the display of thevirtual and/or real images.

Further, although an example in which the display 26 is mounted as apart of the mobile terminal 20-1 is shown in FIG. 4, the display 26 maybe a separate member from the mobile terminal 20-1. Further, the display26 may be HMD (Head Mounted Display) that is mounted on the head of auser.

The touch panel 27 is an operation detection unit that is mounted ontothe display 26. The touch panel 27 can detect the proximity or contactof an operating body such as a user's finger or a touch pen. Note thatthe touch panel 27 is just an example of an operation detection unit,and the mobile terminal 20-1 may include another component foroperation, such as a keyboard and a button.

The imaging device 213 may include an imaging optical system and asignal converter as described above with reference to FIG. 3, andcaptures real space images (moving images or still images) according tocontrol by the control unit 241. Note that the imaging device 213 mayinclude a component for imaging of moving images and a component forimaging of still images separately.

The feature value DB 220 is a database that stores feature value data ofimages of real objects. Specifically, in the feature value DB 220,identification information for identifying each real object and featurevalue data of each real object are associated with each other. Thefeature value data may be a set of feature values determined fromlearning images of real objects using SIFT or Random Ferns, for example.

The image recognition unit 224 may recognize a real object contained inthe real space image captured by the imaging device 213, and theposition and posture of the real object in the real space image. Forexample, the image recognition unit 224 recognizes a real objectcontained in the real space image by checking a feature value determinedfrom the real space image against the feature value of each real objectstored in the feature value DB 220. Specifically, the image recognitionunit 224 may determine a feature value of a real object in the realspace image according to a feature value determination method such asSIFT or Random Ferns, and checks the determined feature value againstthe feature value of each real object stored in the database 220. Then,the image recognition unit 224 recognizes the identification informationof the real object associated with the feature value which most matchesthe feature value of the real object in the real space image, and theposition and posture of the real object in the real space image.

Note that the image recognition unit 224 may recognize a real objectsuch as, for example, a known figure or symbol, an artificial marker(e.g. a bar code or a QR code) or a natural marker and recognize theposition and posture of the real object based on the size and shape ofthe real object in the real space image.

Further, although an example in which a real object contained in a realspace image is recognized by image processing is described above, amethod of recognizing a real object is not limited to the imageprocessing. For example, it is feasible to detect a direction towardwhich the imaging device 213 is pointed and the current position of themobile terminal 20-1, and estimate a real object contained in a realspace image and the position and posture of the real object in the realspace image based on a detection result.

The virtual object DB 228 is a database that stores virtual objects tobe superimposed onto real objects. Specifically, in the virtual objectDB 228, identification information of a virtual object, identificationinformation of a real object, setting information indicating a displayposition relative to the real object and a posture, and a virtual objectto be presented to a user are associated with one another. The virtualobject may be in text format or image format.

The virtual object acquisition unit 232 may acquire the virtual objectand the setting information corresponding to the real object recognizedby the image recognition unit 224. For example, the virtual objectacquisition unit 232 acquires the virtual object and the settinginformation associated with the identification information of the realobject recognized by the image recognition unit 224 from the virtualobject DB 228.

The control unit 241 controls the overall operation of the mobileterminal 20-1. For example, the control unit 241 causes the operatingmode of the mobile terminal 20-1 to transition among a plurality ofoperating modes including real time recognition mode and still imageoperation mode according to predetermined criteria for transition.

The real time recognition mode is an operating mode that may recognize areal object contained in a moving image of a real space captured by theimaging device 213 in real time and creates the real-time recognitionscreen by superimposing a virtual object onto the real object. On theother hand, the still image operation mode is an operating mode thatcreates the still image operation screen by superimposing a virtualobject onto a still image captured by the imaging device 213. Note thatthe control unit 241 can change display of the virtual object based on auser operation in any operating mode. Specifically, the control unit 241can move, zoom in/out or scroll a virtual object based on a useroperation. The above-described still image in the still image operationmode may be, for example, one frame that constitutes a moving imagecaptured by the component for imaging of moving images or a still imagecaptured by the component for imaging of still images. Further, theabove-described still image may be one still image composed of aplurality of still images captured. For example, when the mobileterminal 20-1 has a function of combining a plurality of still imagesinto a magnified still image (such as “Swing Panorama” function thatcombines a plurality of still images captured with the direction of theimaging device shifted in steps into one panorama picture), the mobileterminal 20-1 can superimpose a virtual object onto one magnified stillimage in the still image operation mode. In this case, a user operationcan be also performed on one magnified still image. Further, the mobileterminal 20-1 may display the still image which has been captured mostrecently on the initial still image operation screen, and when a useroperation indicates the direction which opposes to the motion of theimaging device (such as drag to the direction opposite to the swingdirection), further display the past still image (which is spatiallyadjacent) with a wider angle of view in addition to the still imagebeing displayed. In this case, it is feasible to move a virtual objectfrom the region of the most recent still image to the region of the paststill image according to a user operation. The synthesis of themagnified still image may be made based on the analysis of the contentsof each image or based on the position and posture of the imaging devicemeasured by a sensor at time of capturing each image.

Layout Examples of Real-Time Recognition Screen and Still ImageOperation Screen

Layouts of the real-time recognition screen and the still imageoperation screen are described hereinafter with reference to FIG. 5. Asshown in FIG. 5, a real-time recognition screen P1 which is created inthe real time recognition mode contains display of the station board 40,which is a real object, and the time table 52, which is a virtual objectsuperimposed onto the station board 40. In order to maintain thereal-time recognition screen P1, a user may maintain pointing theimaging direction of the mobile terminal 20-1 toward the station board40 as shown in FIG. 5.

On the other hand, a still image operation screen P2 which is created inthe still image operation mode contains display of the station board 40,which is a real object, the time table 52, which is a virtual objectsuperimposed onto the station board 40, a tab “Time Table” 62, a tab“Area Map” 64, and a return key 66.

The control unit 241 may change the virtual object to be displayedaccording to a user operation on the tab “Time Table” 62, the tab “AreaMap” 64 and the like. For example, when the tab “Area Map” 64 isselected by a user, the control unit 241 changes the virtual object tobe superimposed onto the station board 40 from the time table 52 to thearea map.

In the still image operation mode, the still image operation screen P2containing the virtual object can be displayed without a need for a userto point the imaging direction of the mobile terminal 20-1 toward thereal object as shown in FIG. 5. It is thereby possible to improve theusability of the mobile terminal 20-1 for the user and the operabilityof display of the virtual object.

Note that the return key 66 is an example of a transition operationbutton, and, when the return key 66 is entered by a user, for example,the control unit 241 causes the operating mode to transition from thestill image operation mode to the real time recognition mode.

FIG. 6 is an explaining embodiment view showing an alternative exampleof the still image operation screen. A still image operation screen P3according to an alternative example contains a plurality of thumbnails(S1, S2, S3, . . . ), a return key 66 and additional information 68 foreach thumbnail. The control unit 241 can accept a thumbnail selectionoperation on the still image operation screen P3 and display theselected thumbnail as a main item. Note that the plurality of thumbnailsmay be a plurality of frames that constitute the moving image obtainedin the real time recognition mode. Further, each thumbnail may be acomposite image of a still image of the real space and a virtual object,or a still image of the real space not containing a virtual object.

Criteria for Transition of Operating Mode

Transition criteria for the control unit 241 to determine whether tocause the operating mode to transition or not are described hereinafter.

First, criteria for transition from the real time recognition mode tothe still image operation mode may be any one of the following criteriaor a combination of the following criteria.

(1-1) A real object is recognized by the image recognition unit 224 inthe real time recognition mode

(1-2) An operating body such as a user's finger comes into contact withthe touch panel 27.

(1-3) An operating body is removed from the touch panel 27.

(1-4) Contact or removal of an operating body with or from the touchpanel 27 is detected within a judgment range of a real object.

Accordingly, the control unit 241 may cause the operating mode totransition to the still image operation mode when a real object isrecognized by the image recognition unit 224 and contact of an operatingbody with the touch panel 27 is detected within the judgment range ofthe real object, for example. Note that, as an additional explanation ofthe judgment range of the real object, the judgment range of the realobject is a range related to a recognition range of the real object,which may be a range including the recognition range of the real object,or a part of the recognition range of the real object. When a userconducts an operation within the judgment range of the real object, thecontrol unit 241 judges that the real object is selected even if it isoutside the recognition range of the real object.

Note that the control unit 241 may use a real space image which isobtained at a point when the criteria for transition to the still imageoperation mode are met as a still image of the still image operationscreen or may use a real space image which is obtained after thecriteria for transition to the still image operation mode are met as astill image of the still image operation screen. In the latter case, thecontrol unit 241 may direct the imaging device 213 to capture a stillimage using the component for imaging of still images.

Next, criteria for transition from the still image operation mode to thereal time recognition mode may be any one of the following criteria or acombination of the following criteria.

(2-1) An operating body such as a user's finger comes into contact withthe touch panel 27.

(2-2) An operating body is removed from the touch panel 27.

(2-3) Contact or removal of an operating body with or from the touchpanel 27 is detected outside a judgment range of a real object.

(2-4) The return key 66 is entered.

Accordingly, the control unit 241 may cause the operating mode totransition to the real time recognition mode when contact of anoperating body with the touch panel 27 is detected outside the judgmentrange of the real object, for example.

As described above, because the mobile terminal 20-1 according to thefirst embodiment has the still image operation mode, it is possible todisplay the still image operation screen in which a virtual object issuperimposed onto a still image of the real space on the display 26without a need for a user to keep pointing the imaging direction of themobile terminal 20-1 toward the real object. It is thereby possible toimprove the usability of the mobile terminal 20-1 for the user and theoperability of display of the virtual object.

(Operation of Mobile Terminal)

An operation of the mobile terminal 20-1 according to the firstembodiment is described hereinafter with reference to FIGS. 7 and 8. Asshown in FIGS. 7 and 8, the mobile terminal 20-1 can execute therecognition process and the user IF process in a parallel fashion.

FIG. 7 is a flowchart showing an operation of the mobile terminal 20-1in the real-time recognition mode. As shown in FIG. 7, in the real timerecognition mode, the image recognition unit 224 first determineswhether a real space image which is input from the imaging device 213 isupdated or not (S304). When the real space image is updated, the imagerecognition unit 224 attempts to recognize a real object contained inthe real space image and recognize the posture of the real object(S308). When the real object is recognized in S308, the virtual objectacquisition unit 232 acquires a virtual object corresponding to the realobject from the virtual object DB 228, and the control unit 241generates display of the virtual object acquired by the virtual objectacquisition unit 232 (S316). Note that, when it is determined that thereal space image is not updated (NO in S304), when the real object isnot recognized in S308 (NO in S312), or after the processing of S316,the processing from S304 is repeated.

On the other hand, when it is determined that the real space image isupdated in S304, the control unit 241 updates the real space image inthe real-time recognition screen (S404). Further, when display of thevirtual object is generated in S316, the control unit 241 may updatedisplay of the virtual object in the real-time recognition screen(S408). Then, when the criteria for transition to the still imageoperation mode are met, the control unit 241 causes the operating modeof the mobile terminal 20-1 to transition to the still image operationmode (S412).

FIG. 8 is a flowchart showing an operation of the mobile terminal 20-1in the still image operation mode. As shown in FIG. 8, in the stillimage operation mode, the control unit 241 first acquires a still image(S320), and reads posture information of the real object recognized inS308 regarding the still image (S324). Note that, in the case of newlycapturing a still image for the still image operation screen,recognition of the posture of a real object contained in the new stillimage may be performed. Then, the control unit 241 generates display ofa virtual object corresponding to the real object which is acquired bythe virtual object acquisition unit 232 (S328).

On the other hand, when a still image is acquired in S320 (S420), thecontrol unit 241 displays the still image on the display 26 (S424).Further, the control unit 241 displays the display of the virtual objectgenerated in S328 on the display 26 (S428). Then, when the criteria fortransition to the real time recognition mode are met, the control unit241 causes the operating mode of the mobile terminal 20-1 to transitionto the real time recognition mode (S432). On the other hand, when a useroperation which is not an operation to indicate mode transition isdetected, the control unit 241 changes the internal state of anapplication according to the user operation (S436) and repeats theprocessing from S424.

2-2. Second Embodiment

Hereinafter, a second embodiment of the present disclosure is described.Although an explicit operation such as contact or removal of a user'sfinger with or from the touch panel 27 is described as the criteria fortransition of the operating mode in the first embodiment, other criteriafor transition of the operating mode are proposed in the secondembodiment.

FIG. 9 is a functional block diagram showing a configuration of a mobileterminal 20-2 according to the second embodiment. As shown in FIG. 9,the mobile terminal 20-2 according to the second embodiment includes adisplay 26, a touch panel 27, a motion sensor 28, an imaging device 213,a feature value DB 220, an image recognition unit 224, a virtual objectDB 228, a virtual object acquisition unit 232, and a control unit 242.The functions of the display 26, the touch panel 27, the imaging device213, the feature value DB 220, the image recognition unit 224, thevirtual object DB 228, and the virtual object acquisition unit 232 arethe same as those described in the first embodiment, and the elementsdifferent from those in the first embodiment are mainly described below.

The motion sensor 28 is a sensor that detects the motion of the mobileterminal 20-2. For example, the motion sensor 28 detects accelerationsin the X-axis direction, the Y-axis direction and the Z-axis direction(Gx, Gy, Gz) which may be orthogonal to one another. Note that, however,the target of detection of the motion sensor 28 is not limited toaccelerations in three dimensions. For example, when the relativepositions of the imaging device 213 and the motion sensor 28 are known,the target of detection of the motion sensor 28 may be an accelerationin one dimension or accelerations in two dimensions. Further, the motionsensor 28 may be a gyro sensor.

The control unit 242 has a function of controlling the transition of theoperating mode based on the detection result of the motion by the motionsensor 28, in addition to all or some of the functions of the controlunit 241 according to the first embodiment. This is described below.

In use a user may hold up the mobile terminal 20-2 toward the realobject as shown on the left of FIG. 5 in the real-time recognition mode.On the other hand, in the still image operation mode, a user may holdthe mobile terminal 20-2 at the correct position with respect to thedownward line of sight of a user as shown on the right of FIG. 5.Specifically, the position and posture of the mobile terminal 20-2 inthe real-time recognition mode and the position and posture of themobile terminal 20-2 in the still image operation mode may be different.

From this point of view, in the second embodiment, a motion to be givento the mobile terminal 20-2 by a user at the time of mode transition isperceived as a mode transition operation, and the control unit 242causes the operating mode of the mobile terminal 20-2 to transitionbased on the detection result of a motion by the motion sensor 28.

For example, when each or one or more of the amplitude components of themotion detection result (Gx, Gy, Gz) by the motion sensor 28 varies bythe value more than a set value within a predetermined period of time,the control unit 242 may cause the operating mode of the mobile terminal20-2 to transition.

Alternatively, the control unit 242 may cause the operating mode totransition from the real time recognition mode to the still imageoperation mode when a first motion pattern is detected by the motionsensor 28, and cause the operating mode to transition from the stillimage operation mode to the real time recognition mode when a secondmotion pattern different from the first motion pattern is detected bythe motion sensor 28. The first motion pattern may be a pattern in whichthe mobile terminal 20-2 may be moved from the state of being held up tothe state of being at the correct position with respect to the downwardline of sight of a user in accordance with actual use case. Further, thesecond motion pattern may be the reverse motion pattern of the firstmotion pattern.

However, the motion patterns serving as criteria for transition are notlimited to the above example, and various motion patterns may be definedas criteria for transition. For example, various motion patternsincluding motion patterns in which the mobile terminal 20-2 is swung,rotated and the like can be defined as criteria for transition.

Further, the control unit 242 may control the transition of theoperating mode according to the AND or OR of the above-describedcriteria for transition related to a motion and the criteria fortransition described in the first embodiment. For example, the controlunit 242 may cause the operating mode to transition to the still imageoperation mode when a user's finger comes into contact with the touchpanel 27 (1-2) and the first motion pattern is detected by the motionsensor 28. Likewise, the control unit 242 may cause the operating modeto transition to the real time recognition mode when a user's fingercomes into contact with the touch panel 27 (2-1) and the second motionpattern is detected by the motion sensor 28.

As described above, according to the second embodiment, the motion whichis expected to be given to the mobile terminal 20-2 by a user at thetime of mode transition is used as criteria for transition, therebyreducing the load of a user operation for mode transition.

Note that there may be a case where a real object which attracts user'sinterest is not contained in the real space image captured at the timewhen a motion which meets criteria for transition is detected by themotion sensor 28. In light of this, the mobile terminal 20-2 accordingto the second embodiment may store the real space image captured duringa predetermined period in the past and use the real space image before amotion which meets criteria for transition is detected by the motionsensor 28 as a still image for the still image operation screen.Particularly, it is effective to use the real space image at the startof the motion pattern as a still image for the still image operationscreen.

2-3. Third Embodiment

Hereinafter, a third embodiment of the present disclosure is described.In the third embodiment of the disclosure, a configuration for using ahigher quality still image for the still image operation screen isproposed.

FIG. 10 is a functional block diagram showing a configuration of amobile terminal 20-3 according to the third embodiment. As shown in FIG.10, the mobile terminal 20-3 according to the third embodiment includesa display 26, a touch panel 27, a motion sensor 28, an imaging device213, a feature value DB 220, an image recognition unit 224, a virtualobject DB 228, a virtual object acquisition unit 232, and a control unit243, and a blur measurement unit 250. The functions of the display 26,the touch panel 27, the motion sensor 28, the imaging device 213, thefeature value DB 220, the image recognition unit 224, the virtual objectDB 228, and the virtual object acquisition unit 232 are the same asthose described in the second embodiment, and the elements differentfrom those in the second embodiment are mainly described below.

The blur measurement unit 250 is a quality measurement unit that buffersmoving images of the real space captured by the imaging device 213 inthe real time recognition mode and detects a blur of each of the past Nframes at the transition from the real time recognition mode to thestill image operation mode. For example, the blur measurement unit 250may transform image data of each frame into a frequency component bydiscrete cosine transform and measure the blur of each frame from thefrequency component of the image data.

The control unit 243 has a function of selecting a still image for thestill image operation screen, in addition to all or some of thefunctions of the control unit 242 according to the second embodiment.For example, the control unit 243 may select a frame with the least bluramong the past N frames based on the measurement result by the blurmeasurement unit 250 and uses the selected frame as a still image forthe still image operation screen. This configuration enables display ofa still image with good quality on the still image operation screen.

Note that the mobile terminal 20-3 may store a real object and theposture of the real object which have been recognized for each frame bythe image recognition unit 224 as a recognition result of each frame,and the control unit 243 may generate the still image operation screenusing the recognition result stored for the selected frame.

2-4. Fourth Embodiment

Hereinafter, a fourth embodiment of the present disclosure is described.Although an example in which a mobile terminal implements AR alone isdescribed in the first to third embodiments, a mobile terminal and aserver installed on the network side may implement AR in combination.

FIG. 11 is a view showing an AR system according to the fourthembodiment. As shown in FIG. 11, the AR system according to the fourthembodiment includes a mobile terminal 20-4, a network 12, an AR server30, and a station board 40, which is an example of a real object.Configurations of the mobile terminal 20-4 and the AR server 30 aredescribed specifically with reference to FIG. 12.

FIG. 12 is a block diagram showing configurations of the mobile terminal20-4 and the AR server 30 according to the fourth embodiment. As shownin FIG. 12, the mobile terminal 20-4 according to the fourth embodimentincludes a display 26, a touch panel 27, a motion sensor 28, an imagingdevice 213, a feature value DB 220, an image recognition unit 224, acontrol unit 244, and a communication unit 260. The display 26, thetouch panel 27, the motion sensor 28, the imaging device 213, thefeature value DB 220 and the image recognition unit 224 are the same asthose described in the first to third embodiments, and the elementsdifferent from those in the first to third embodiments are mainlydescribed below.

The communication unit 260 is an interface with the AR server 30, and itmay act as a transmitting unit that transmits information to the ARserver 30 and a receiving unit that receives information from the ARserver 30. For example, the communication unit 260 requests the ARserver 30 to transmit a virtual object corresponding to the real objectrecognized by the image recognition unit 224. The communication unit 260may then transmit identification information of the real objectrecognized by the image recognition unit 224 to the AR server 30. Then,the communication unit 260 receives the virtual object transmitted fromthe AR server 30 in response to the above request.

The control unit 244 has a function of superimposing the virtual objectreceived by the communication unit 260 onto a still image and creatingthe still image operation screen, in addition to all or some of thefunctions of the control unit 242 according to the second embodiment.

On the other hand, the AR server 30 includes a virtual objectacquisition unit 332, a virtual object DB 328, and a communication unit360 as shown in FIG. 12.

The virtual object DB 328 may include database that stores virtualobjects to be superimposed onto real objects. Specifically, in thevirtual object DB 328, identification information of a virtual object,identification information of a real object, setting informationindicating a display position relative to the real object and a posture,and a virtual object to be presented to a user are associated with oneanother. The virtual object may be in text format or image format.

The virtual object acquisition unit 332 acquires the virtual object andthe setting information associated with the identification informationof the real object received by the communication unit 360 from thevirtual object DB 328.

The communication unit 360 may include an interface with the mobileterminal 20-4, and act as a transmitting unit that transmits informationto the mobile terminal 20-4 and a receiving unit that receivesinformation from the mobile terminal 20-4. For example, thecommunication unit 360 receives identification information of a realobject from the mobile terminal 20-4 as a request for transmission of avirtual object, and transmits the virtual object acquired by the virtualobject acquisition unit 332 and setting information to the mobileterminal 20-4.

As described above, according to the fourth embodiment, cloud computingcan be implemented by incorporating some of the functions for AR intothe AR server 30. Note that, although an example in which the functionsof the virtual object DB and the virtual object acquisition unit areincorporated into the AR server 30 is described in the fourthembodiment, the functions to be incorporated into the AR server 30 arenot limited thereto. For example, the functions of the image recognitionunit and the feature value DB may be also incorporated into the ARserver 30. In this case, the mobile terminal 20-4 transmits an imagecaptured by the imaging device 213 to the AR server 30, and the ARserver 30 performs recognition of a real object and acquisition of avirtual object and then transmits the virtual object to the mobileterminal 20-4.

Furthermore, the functions of creating the real-time recognition screenand the still image operation screen may be incorporated into the ARserver 30. In this case, the mobile terminal 20-4 transmits an imagecaptured by the imaging device 213 to the AR server 30, and the ARserver 30 performs recognition of a real object, acquisition of avirtual object and superimposition of the virtual object, and thentransmits the real-time recognition screen or the still image operationscreen in which the virtual object is superimposed to the mobileterminal 20-4. In the case of incorporating the functions of creatingthe real-time recognition screen and the still image operation screeninto the AR server 30, the AR server 30 serves as a display controldevice.

In the above-described AR system, when the mobile terminal 20-4 requeststhe AR server 30 to transmit a virtual object at the time of transitionto the still image operation mode, there is a concern on the occurrenceof a latency from reception to display of the virtual object. In lightof this, when criteria for transition to the still image operation modeare likely to be met, the mobile terminal 20-4 according to the fourthembodiment requests the AR server 30 to transmit a virtual object beforethe criteria for transition to the still image operation mode are met.This is described specifically with reference to FIGS. 13 to 15.

First Operation Example

FIG. 13 is a sequence chart showing a first operation example of thefourth embodiment. Note that, in the first operation example, assume thecase where the criteria for transition from the real time recognitionmode to the still image operation mode are that a user taps the touchpanel 27, e.g., that a user's finger comes into contact with the touchpanel 27 and is then removed from the touch panel 27.

As shown in FIG. 13, when contact of a user's finger with the touchpanel 27 is detected in the real time recognition mode (S504), thecommunication unit 260 requests the AR server 30 to transmit a virtualobject prior to transition to the still image operation mode (S508).

The virtual object acquisition unit 332 of the AR server 30 acquires avirtual object from the virtual object DB 328 in response to the requestfrom the mobile terminal 20-4 (S512), and the communication unit 360transmits the virtual object acquired by the virtual object acquisitionunit 332 to the mobile terminal 20-4 (S516).

When removal of a user's finger from the touch panel 27 is detected(S520), the control unit 244 of the mobile terminal 20-4 makestransition to the still image operation mode and displays the stillimage operation screen containing the virtual object on the display 26(S524).

As described above, when a part of the user operation for transition tothe still image operation mode is detected, the mobile terminal 20-4requests the transmission of a virtual object prior to transition to thestill image operation mode. This configuration allows the still imageoperation screen to be displayed smoothly at the transition to the stillimage operation mode. Note that, when a latency for display of the stillimage operation screen occurs in spite of application of this operationexample, the control unit 244 may display “Loading” on the display.

Second Operation Example

FIG. 14 is a sequence chart showing a second operation example of thefourth embodiment. Note that, in the second operation example, assumethe case where the criteria for transition from the real timerecognition mode to the still image operation mode are that a realobject is recognized by the image recognition unit 224 in the real timerecognition mode and that a user taps the touch panel 27.

As shown in FIG. 14, when a real object is recognized by the imagerecognition unit 224 in the real time recognition mode (S604), thecommunication unit 260 may request the AR server 30 to transmit avirtual object prior to transition to the still image operation mode(S608).

The virtual object acquisition unit 332 of the AR server 30 may acquirea virtual object from the virtual object DB 328 in response to therequest from the mobile terminal 20-4 (S612), and the communication unit360 transmits the virtual object acquired by the virtual objectacquisition unit 332 to the mobile terminal 20-4 (S616).

When the touch panel 27 is tapped by a user (S620), the control unit 244of the mobile terminal 20-4 makes transition to the still imageoperation mode and displays the still image operation screen containingthe virtual object on the display 26 (S624).

As described above, when a part of the criteria for transition to thestill image operation mode is met, the mobile terminal 20-4 may requestthe transmission of a virtual object prior to transition to the stillimage operation mode. This configuration allows the still imageoperation screen to be displayed smoothly at the transition to the stillimage operation mode, just like the first operation example.

Third Operation Example

FIG. 15 is a chart showing a third operation example of the fourthembodiment. Note that, in the third operation example, assume the casewhere the criteria for transition from the real time recognition mode tothe still image operation mode are that a user's finger comes intocontact with the touch panel 27 and that a change in the posture of themobile terminal 20-4 is detected.

As shown in FIG. 15, when contact of a user's finger with the touchpanel 27 is detected (S704), the communication unit 260 requests the ARserver 30 to transmit a virtual object prior to transition to the stillimage operation mode (S708).

The virtual object acquisition unit 332 of the AR server 30 acquires avirtual object from the virtual object DB 328 in response to the requestfrom the mobile terminal 20-4 (S712), and the communication unit 360transmits the virtual object acquired by the virtual object acquisitionunit 332 to the mobile terminal 20-4 (S716).

When a change in the posture (a predetermined motion pattern) of themobile terminal 20-4 is detected by the motion sensor 28 (S720), thecontrol unit 244 of the mobile terminal 20-4 makes transition to thestill image operation mode and displays the still image operation screencontaining the virtual object on the display 26 (S724).

In the above third operation example, as in the second operationexample, it is possible to display the still image operation screensmoothly at the transition to the still image operation mode.

3. Summary

As described above, because the mobile terminal 20 according to eachembodiment of the present disclosure may have the still image operationmode, it is possible to display the still image operation screen inwhich a virtual object is superimposed onto a still image of the realspace on the display 26 without a need for a user to keep pointing theimaging direction of the mobile terminal 20 toward the real object. Itis thereby possible to improve the usability of the mobile terminal 20for the user and the operability of display of the virtual object.

Although preferred embodiments of the present disclosure are describedabove with reference to the appended drawings, the present disclosure isnot limited thereto. It should be understood by those skilled in the artthat various modifications, combinations, sub-combinations andalterations may occur depending on design requirements and other factorsinsofar as they are within the scope of the appended claims or theequivalents thereof.

For example, although an operation on the touch panel 27 which isdetected by the touch panel 27 and a motion of the mobile terminal 20which is detected by the motion sensor 28 are described as examples ofdetection of a user operation serving as a trigger for transition to thestill image operation mode, a user operation is not limited thereto.Another example of detection of a user operation is recognition of auser gesture. A user gesture can be recognized based on an imageacquired by the imaging device 213 or recognized based on an imageacquired by another imaging system. Note that the imaging device 213 oranother imaging system may image a user gesture by a function such as aninfrared camera or a Depth camera.

Further, although an example in which the display control device is themobile terminal 20 is mainly described in the above embodiments, thedisplay control device may be a device larger than the mobile terminal20, such as a television or a display device. For example, by using alarge display capable of displaying the entire body of a user withconnection or integration of an imaging system that images the user fromthe display control device side with or into the display control device,it is possible to build a function like a mirror that reflects the userand thereby implement an AR application that superimposes a virtualobject onto the user to allow operation of the virtual object.

Further, it is not always necessary to perform the steps in theprocessing of the mobile terminal 20 or the AR server 30 in thisspecification in chronological order according to the sequence shown inthe sequence charts or the flowcharts. For example, the steps in theprocessing of the mobile terminal 20 or the AR server 30 may beprocessed in a sequence different from the sequence shown in theflowchart or processed in parallel.

Furthermore, it is feasible to create a computer program that causeshardware such a CPU, ROM and RAM incorporated into the mobile terminal20 or the AR server 30 to perform the equal functions to the elements ofthe mobile terminal 20 or the AR server 30 described above. Further, astorage medium in which such a computer program is stored may be alsoprovided.

Further, the following configurations are also within the scope of thepresent disclosure.

(1)

A display control device including a control unit that causes display ofa display device to transition between a real space moving imagecontaining a moving image of a real space and a composite imagecontaining a still image of the real space and a virtual objectcorresponding to a real object contained in the still image.

(2)

The display control device according to (1), wherein the real spacemoving image contains a moving image of a real space being imaged by animaging device.

(3)

The display control device according to (1) or (2), wherein display ofthe virtual object contained in the composite image varies by a useroperation.

(4)

The display control device according to any one of (1) to (3), furtherincluding a recognition unit that recognizes the real object in the realspace moving image, wherein the control unit causes display of thedisplay device to transition from the real space moving image to thecomposite image when the real object is recognized by the recognitionunit.

(5)

The display control device according to any one of (1) to (4), furtherincluding an operation detection unit that detects a user operation,wherein the control unit causes display of the display device totransition when a predetermined user operation is detected by theoperation detection unit.

(6)

The display control device according to (5), wherein the operationdetection unit detects the predetermined user operation on a touch panelmounted on the display device.

(7)

The display control device according to (6), wherein the control unitcauses display of the display device to transition from the real spacemoving image to the composite image when a user operation within ajudgment range of the real object on the touch panel is detected.

(8)

The display control device according to (7), wherein the control unitcauses display of the display device to transition from the compositeimage to the real space moving image when a user operation outside thejudgment range of the real object on the touch panel is detected.

(9)

The display control device according to (8), wherein the composite imagecontains a transition operation button, and wherein the control unitcauses display of the display device to transition from the compositeimage to the real space moving image when a user operation in thetransition operation button on the touch panel is detected.

(10)

The display control device according to any one of (5) to (9), whereinthe operation detection unit includes a motion sensor that detects amotion of the display control device, and wherein the control unitcauses display of the display device to transition from the real spacemoving image to the composite image when a first motion pattern isdetected by the motion sensor.

(11)

The display control device according to (10), wherein the control unitcauses display of the display device to transition from the compositeimage to the real space moving image when a second motion patterndifferent from the first motion pattern is detected by the motionsensor.

(12)

The display control device according to any one of (1) to (11), furtherincluding a communication unit that communicates with a server on anetwork, wherein the communication unit receives the virtual objectcorresponding to the real object from the server.

(13)

The display control device according to (12), wherein the communicationunit requests the server to transmit the virtual object when a part ofthe predetermined user operation is detected by the operation detectionunit, and wherein the control unit causes display of the display deviceto transition to the composite image after a whole of the predetermineduser operation is detected by the operation detection unit.

(14)

The display control device according to (12), wherein the communicationunit requests the server to transmit the virtual object when the realobject is recognized by the recognition unit, and wherein the controlunit causes display of the display device to transition to the compositeimage after the predetermined user operation is detected by theoperation detection unit.

(15)

The display control device according to any one of (1) to (14), furtherincluding a quality measurement unit that measures a quality of eachframe constituting the real space moving image, wherein the control unitselects one frame constituting the real space moving image as the stillimage to be contained in the composite image based on a measurementresult by the quality measurement unit.

(16)

A display control method including causing display of a display deviceto transition between a real space moving image containing a movingimage of a real space and a composite image containing a still image ofthe real space and a virtual object corresponding to a real objectcontained in the still image.

(17)

A program causing a computer to function as a display control deviceincluding a control unit that causes display of a display device totransition between a real space moving image containing a moving imageof a real space and a composite image containing a still image of thereal space and a virtual object corresponding to a real object containedin the still image.

(18)

The program according to (17), wherein the real space moving imagecontains a moving image of a real space being imaged by an imagingdevice.

(19)

The program according to (17) or (18), wherein display of the virtualobject contained in the composite image varies by a user operation.

(20)

The program according to any one of (17) to (19), wherein the displaycontrol device further includes a recognition unit that recognizes thereal object in the real space moving image, and wherein the control unitcauses display of the display device to transition from the real spacemoving image to the composite image when the real object is recognizedby the recognition unit.

The invention claimed is:
 1. An apparatus comprising: a memory storinginstructions; an imaging device to capture a number of live images inreal space; and a control unit configured to operate in a real timerecognition mode and a still image operation mode and with regardthereto to execute the instructions to: detect an object of interestwithin a real space image; obtain a still image of the object byselecting the still image from among the number of live images of realspace captured by the imaging device; create a real time recognitionscreen, when operating in the real time recognition mode, in which avirtual image relating to the object is superimposed on a live image ofthe real space, and in which to maintain the created real timerecognition screen the live image is continuously captured bymaintaining the imaging device in a same imaging direction so as tocapture the object; create a still image screen, when operating in thestill image operation mode, in which the virtual image relating to theobject is superimposed on the still image, and in which the createdstill image screen is maintainable without maintaining the imagingdevice in the constant imaging direction; and cause the real timeoperation mode to be transitioned to the still image operation mode uponoccurrence of a first predetermined criteria so that the real timerecognition screen is switched to the still image screen upon theoccurrence of the first predetermined criteria, and cause the stillimage operation mode to be transitioned to the real time operation modeupon occurrence of a second predetermined criteria so that the stillimage screen is switched to the real time recognition screen upon theoccurrence of the second predetermined criteria which is different fromthe first predetermined criteria, in which the first predeterminedcriteria is one of a number of first conditions which includes theapparatus being moved in a first motion pattern, and in which the secondpredetermined criteria is one of a number of second conditions whichincludes the apparatus being moved in a second motion pattern which isdifferent from the first motion pattern.
 2. The apparatus of claim 1,wherein the virtual image comprises at least one interactive userfunction.
 3. The apparatus of claim 1, comprising instructions to obtainthe still image in response to a user input.
 4. The apparatus of claim3, wherein the user input is an input on a touch panel.
 5. The apparatusof claim 1, comprising instructions to obtain the virtual image from aremote server.
 6. The apparatus of claim 1, wherein obtaining the stillimage comprises obtaining the still image from a remote server.
 7. Amethod for use with an apparatus having an imaging device to capture anumber of live images in real space, said method comprising: detectingan object of interest within an image; obtaining a still image of theobject by selecting the still image from among the number of live imagesof real space captured by the imaging device; creating a real timerecognition screen, when operating in a real time recognition mode, inwhich a virtual image relating to the object is superimposed on a liveimage of the real space, and in which to maintain the created real timerecognition screen the live image is continuously captured bymaintaining the imaging device in a same imaging direction so as tocapture the object; creating a still image screen, when operating in astill image operation mode, in which the virtual image relating to theobject is superimposed on the still image, and in which the createdstill image screen is maintainable without maintaining the imagingdevice in the constant imaging direction; and causing the real timeoperation mode to be transitioned to the still image operation mode uponoccurrence of a first predetermined criteria so that the real timerecognition screen is switched to the still image screen upon theoccurrence of the first predetermined criteria, and causing the stillimage operation mode to be transitioned to the real time operation modeupon occurrence of a second predetermined criteria so that the stillimage screen is switched to the real time recognition screen upon theoccurrence of the second predetermined criteria which is different fromthe first predetermined criteria, in which the first predeterminedcriteria is one of a number of first conditions which includes theapparatus being moved in a first motion pattern, and in which the secondpredetermined criteria is one of a number of second conditions whichincludes the apparatus being moved in a second motion pattern which isdifferent from the first motion pattern.
 8. A tangibly embodiednon-transitory computer-readable medium storing instructions which, whenexecuted by a processor of an apparatus having an imaging device tocapture a number of live images in real space, perform a methodcomprising: detecting an object of interest within an image; obtaining astill image of the object by selecting the still image from among thenumber of live images of real space captured by the imaging device;creating a real time recognition screen, when operating in a real timerecognition mode, in which a virtual image relating to the object issuperimposed on a live image of the real space, and in which to maintainthe created real time recognition screen the live image is continuouslycaptured by maintaining the imaging device in a same imaging directionso as to capture the object; creating a still image screen, whenoperating in a still image operation mode, in which the virtual imagerelating to the object is superimposed on the still image, and in whichthe created still image screen is maintainable without maintaining theimaging device in the constant imaging direction; and causing the realtime operation mode to be transitioned to the still image operation modeupon occurrence of a first predetermined criteria so that the real timerecognition screen is switched to the still image screen upon theoccurrence of the first predetermined criteria, and causing the stillimage operation mode to be transitioned to the real time operation modeupon occurrence of a second predetermined criteria so that the stillimage screen is switched to the real time recognition screen upon theoccurrence of the second predetermined criteria which is different fromthe first predetermined criteria, in which the first predeterminedcriteria is one of a number of first conditions which includes theapparatus being moved in a first motion pattern, and in which the secondpredetermined criteria is one of a number of second conditions whichincludes the apparatus being moved in a second motion pattern which isdifferent from the first motion pattern.